Evaluation of edge direction information

ABSTRACT

This invention concerns recognizing and locating a physically demarcated body part by using only a relatively small amount of computation, but still produces a sufficient degree of recognition accuracy. For this purpose a procedure is proposed for detecting physically demarcated body parts (face, hand, leg) of a person&#39;s image ( 5 ) if a body part ( 2 ) is depicted in front of a background ( 3 ). Borderlines ( 5   d   , 5   e ) in the image are only evaluated along line directions ( 5   a   ′, 4   a   ′, 4   b   ′, 5   c ′) to determine, by comparing with model ( 30 ), whether the body part image corresponds to a type of body part given by the model. In addition, line directions ( 5   d   ′, 5   e ′) inside a body part image and borderline directions ( 5   a ) of a physically demarcated body part are used to locate and store its position.

This invention is concerned with the recognition of types of physicalbody parts like faces, hands, legs or any other body parts. These bodypart types have to be detected in static or moving pictures irrespectiveof whether it is known that the required part is present or not. Whensuch a body part is detected, its exact position (its coordinates)should be indicated in the image and its size in terms of the measuringsystem used should also be made available. The procedure must beautomatic.

In recent years the techniques for face recognition have been based onthe typical gray scale intensity of face images. This detectionprocedure as applied to static or moving images is based on a gray scalepicture, but if this term is used below, its meaning may include otherintensities like color pictures or extracts from color pictures, as wellas any other type of pictorial information which involves intensityvariations. If the term “gray scale value” is used below, it should beunderstood in this general sense.

One type of recognition procedure endeavors to detect facial featureslike eyes, nose or mouth independently and then determine theirposition. The individual localities are subsequently used to find theactual face in the image according to specified rules or on the basis ofstatistical models.

The evaluation of information about edge directions and edge clarity hasbeen proposed in the technical literature, see the article byDonahue/Rokhlin on information about edge directions: “On the use oflevel curves in image analysis”, Image Understanding, Vol. 57 Nr 2,1993, pages 185 to 203, especially Paragraphs 2 and 3 dealing withtangent vector calculation, and FIG. 2 in which the vectorrepresentation is illustrated. Elsewhere an operator is proposed whichwould be able to establish edges in digital images, compare Hueckel: “Anoperator which locates edges in digital pictures”, J. Assoc. Comput.,March 1971, Vol. 18, pages 113 to 125. For the purpose of facerecognition edge information (also described as information aboutborderlines) of Maio/Maltoni has been employed, see “Fast face locationin complex backgrounds”, Face recognition, from theory to applications,NATO ASI Series F: Computer and Systems Sciences, Vol. 163, 1998, pages568 to 577, as well as a later publication of the same authors inPattern Recognition, Vol. 33, 2000, pages 1525 to 1539: “Real-time facelocation on gray-scale static images”.

In later publications gray scale images are shown in the respective FIG.2 which have been converted to edge direction images in terms of vectorlengths and consistency, called direction reliability. In those casesthe vector direction represents the tangent to the edge of the image,and the length of the vector, called significance, comprises the sum ofthe contrast values in the sense of edge thickness or edge clarity. Inaddition, “consistency” is used and explained as direction reliability.

The evaluation of numerous pieces of information about direction,reliability, and clarity is complex, and requires a great deal ofcomputing power. Even modern computers cannot provide sufficientcomputing power, and small computers cannot be used.

For this reason the problem has been reformulated to restrict researchto physically separate body parts like faces, which requires decreasedcomputing power but still provides sufficient accuracy of recognition.

This problem solves claim 1 of the endeavor. It is also solved by meansof a procedure along the lines of claims 15 or 23.

A procedure according to claim 16 prepares a detection procedure(recognition) along the lines of claims 1, 15 or 23.

The research is based on evaluating only relevant information instead ofall given information. Only decisive direction information which can bederived from the borderlines of an intensity image, e.g. a monochromeimage or a color extract. The borderlines are often also called “edges”as if they originated from actual edges in a two-dimensional image. Thisedge or line has both a direction and a thickness (clarity). In thepresent state of technology the methods mentioned above can calculatesuch information from monochrome (gray) images. Edge directions aredecisive, but edge clarity is ignored in comparison to a model image.The model image established artificially, represents the type of bodypart found in a monochrome image and located by means of positionalinformation (claim 3 and claim 11).

The difference between detecting and locating can be seen in the factthat additional previous information is available in the latter case,namely that at least one searched for body part is present in the image.Then positional information is made available as well as the location inthe image (determined from brightness values). This is used in the nextstage as target image for finding the body part. For the purpose ofdefining the sought-for body part in the target image, the procedure canalso provide information about its size.

If such previous information about the presence of, for example a face,is not available, then the procedure can determine additionally, bymeans of a threshold value, whether there really is such a sought-forface in the target image (claim 14 and claim 9). In addition, thepreviously determined information about position and size can be madeavailable (claim 3).

The proposed procedure makes fast detection possible and achieves a highdegree of certainty of recognition by using only a little modelinformation and information derived from the edge direction image.

Line directions inside one body part image are used, as well as the linedirections of a physically demarcated body part. The model evaluates theedge direction and also the actual edge of the sought-for body part andthe edge directions inside this border (claim 4). Other edge directionsalso present in the target image are allocated a low similarity indexcompared to the model. The result of all this is that positionalinformation of the target image can be determined by comparing theagreement index of the edge direction information in the target imagewith the edge direction information of the model (claims 13 and 23).

If size information is required in addition to positional information,then the relative sizes of the model and the target image can be changed(claim 12). Furthermore, the target image is compared with the entirefirst size of the model. Subsequently the size of either the model orthe target image is changed, followed by another complete comparisonrun. Several such runs provide a number of sets of agreement indices foreach pair of relative sizes of target and model. Agreement indices withthe best similarity indicate the size of the relevant body part soughtfor in the target image.

If the size of the model is increased stepwise for example, then thebest agreement will be found when this size is essentially the same asthat of the sought-for body part which had to be defined according toits size.

The known procedures for determining edge information by applying, forexample, the Sobel operator to every pixel of the image, provideinformation about both direction and clarity (a pixel can either be thesmallest area of an image, or several pixels grouped together). Clarityinformation is often also called edge thickness. If edge thickness isused for comparison with a threshold value, then only values whichexceed some specified minimum, are retained as direction information inthe image (claim 16).

Information is rejected for those positions where the direction is notreliably recognizable and which should be regarded as noise.

Directional information is only evaluated for positions of the targetimage where the clarity is great (claims 18 and 19). In this procedureonly information which provides the greatest recognition expectation, isretained for comparison with the model which also possesses edgedirection information.

If clarity information is set at a standard value according to claim 17,then the weight of the remaining directional information is too low forcomparison with the model. Then comparison of the remaining workingimage with the model is reduced to comparing directions only; thisresults in fast processing and a more certain recognition in spite ofthe reduced information content.

The model has been mentioned repeatedly; it consists, for example, of alarge volume of directional information obtained from test samples.However, the model may simply be a single image used because of its edgedirections as described above, but then those sections of the imagewhere there is directional noise, are filtered out (claim 16 a.E. in analternative way of understanding). The use of a single test image as amodel is justified if the user of the body part which is implied in themodel, like a face, a hand, or a foot, is processed according to claim16.

The comparison between the model with directional information and theremaining directional information in the target image is done by meansof a similarity analysis which can be an angular comparison of adirectional orientation (claim 13 or claim 23).

An angular comparison can consist of the sum of angle differencescalculated for every point of the model according to its size or forevery pixel of the image and then placed at one position of the targetimage for the entire area of the model (claim 24). If the agreement isclose, then the angle difference for each pixel comparison is small,even zero. The sum of small angles is also small (claim 27). The bestagreement is found where the sum is a minimum for the entire targetimage and for various relative positions of model and target image(claim 13).

It would also be possible to use trigonometric functions of angles orangle differences together with angle differences themselves (claim 24).

If there is no a priori (advance) knowledge of whether the sought-fortype of body part is present or not, then a suitable threshold having arelatively low minimum is prepared (claims 9 and 25). Only when asimilarity is found which lies below this minimum threshold value, canit be deduced that the sought-for type of body part is represented inthe image. This threshold value must not be zero nor maximal, otherwisethe procedure would only determine the minimum of all similarity valuesdistributed over the target image. This value must not forcibly agreewith the position of a sought-for body part, but it only reflects avalue which could even be extremely large (claim 14).

If it is known beforehand that a sought-for body part is present, thenit is not necessary to perform a threshold comparison, since the minimumof the similarity values obtained by comparing various sizerelationships, then automatically indicates the position of thesought-for body part.

When positional data is stored (mostly electronically), then evaluationresults in a set of repeated comparisons with the model image where eachcomparison indicates a different local assignation of the model imageand the borderline information of the target image. An example of adifferent local allocation is a pixel-by-pixel sideways emplacement overthe target image. When a borderline of the model image reaches the edgeof the target image, the scan is lowered by one pixel and again comparedwith the entire row of pixels. These repeated comparisons result in aset of similarity values for each position of the model image. The bestsimilarity in this range can reveal the position where the model imagemost closely agrees with the target image.

If, in addition, various size relationships are employed, it wouldresult in further sets of similarity values. And if the similarityindices are compared vertically, it means that the sets of similarityvalues can also be compared to reliably determine the position and thesize of the demarcated body part.

The sum of similarity data, for example the comparison of angles atevery pixel, produces a similarity value which represents a localallocation. It also represents a size allocation in terms of a sizerelationship.

Similarity data follow from angle comparisons (claim 24) using simpleangle differences, or trigonometric functions of angle differences or ofactual angles and adjoining difference determinations.

If reduced directional information is employed (claim 16), it means thatangle information does not exist for all positions in the model, neitherdoes directional information exist for all positions in the reducedtarget image. Only when angle information for the relevant pixelposition is available in both images, can it contribute to a similarityvalue (claim 26) in the sense of the value of similarity data. Thisfollows from the consideration that where noise is present and angleinformation is blocked out because of a low level of reliability, therecannot really be similarity in regard to the sought-for object.

In addition, reduction of directional information according to claim 16can work with a binary operator applied after comparison with thethreshold value (claim 19). This provides an image pattern which hasonly two states for combined pixels, namely the maximum value or theminimum value. An image representation with gray steps between 0 and 255agrees with both the mentioned edge values. This pattern image issuitable for multiplication with the directional information containedin an intermediate image of equal size to produce the working image. Alldirectional data for which the clarity index lies above the thresholdvalue, are retained, but all data which lie below the threshold, aredeleted.

It is not necessary to add all the similarity data for a relativeposition of the model and the intermediate image to obtain a similarityvalue as a sum (claim 20). But it is sufficient to use the bestagreement, i.e. only a limited number of terms. If, for example, thereare 120 possible values, only the 80 smallest ones (with the bestcorresponding similarity data) are required for the summation.

Further simplification is obtained when only acute angles areconsidered. This requires the conversion of angles larger than 90° toacute angles smaller than 90°. This is done in such a way thatborderlines which make an angle of 90°, produce the smallest agreement.Between 90° and 180° the agreement improves, but from 0° to 90° thesimilarity is worse (claim 21).

The model can be used as a model image and operate with directionalinformation having angles between 0° and 90° (claim 22).

Some examples illustrate and amplify the procedure. Furthermore, theprocedure is explained and extended by means of embodiments, but itshould be pointed out that specially selected examples are describedbelow.

FIG. 1 is a block switched image of a first example of the procedureaimed at the recognition of a face in a grayscale image 5.

FIG. 2 is an alternative execution example also directed at therecognition of a face where additional size information is provided.

FIGS. 2 a up to 2 e are representations of different steps in theprocessing of the grayscale image of FIG. 2 a. Also, FIG. 2 e is an edgedirection image (borderline directional image) which contains all theinformation of the initial image in FIG. 2 a.

FIG. 3 is an edge direction image for which only part of the informationof FIG. 2 e is available. What is more, this representation is invertedwith white directions on a dark background.

FIG. 4

FIG. 5 corresponds to a contrasting representation of FIGS. 2 e and 3,where a similar view having dark directional arrows has been selected.

FIG. 6 is a representation of a similarity calculation for a grayscaleimage where different sizes of the model is employed for purposes ofcomparison.

FIG. 6 a is an enlarged representation of FIG. 6 where a brightnessvalue is allocated to every pixel, symbolizing the degree of agreementof the model with a section of the image corresponding to the size ofthe model. Bright areas indicate low similarity, while dark areasrepresent great similarity.

FIG. 7 is a grayscale image belonging to FIG. 6 where the sought-forbody part has been found as the face in frame 60.

FIG. 8 is an example of a model generated from the ten images of asample. Model 30 contains selected edge direction information.

FIG. 1 is based on a grayscale image 5 as an example of an intensityimage. It shows a face in a head and shoulders representation before abackground. The grayscale image has been used as input to the first stepof the procedure. This step 40 evaluates the image in many small areasand derives edge information. The partial image areas are either singlepixels or groups of pixels comprising a small section of the image. Foreach of these partial areas, especially for each pixel of the grayscaleimage, directional edge information is given as well as thicknessinformation. This can be represented as a vector which has bothdirection and magnitude (length).

The single pixels are symbolically indicated with a P in procedure step40.

The resulting edge-orientated image is compared with model 30 in thenext procedure step 41. This is done by shifting a smaller model overthe image. A similarity index is calculated for each position of themodel on the edge direction image. The model is subsequently moved onepixel to the right and a new similarity index is computed. Allsimilarity indices cover the entire area of the edge direction image andthus also the grayscale image (minus the height and breadth of themodel). The result is that a similarity index is available for everyposition of the model in the grayscale image in step 41 which gives theagreement of the model with the edge direction image at that locality.By using these similarity indices obtained in step 41, procedure 60determines the area where the agreement is greatest. This area thenindicates the position in the grayscale image of the sought-for bodypart, a face, for example.

Another example of the procedure is FIG. 2 which is a modified versionof FIG. 1. Size information is provided in addition to the positionalinformation of step 60. Function block 41 corresponds to step 41 of FIG.1, and model 30 corresponds to model 30 of FIG. 1. Grayscale image 5corresponds to image 5 of FIG. 1. This grayscale image is also convertedfrom FIG. 1 to an edge direction image according to step 40. This latterimage is subsequently passed on to step 41 which makes the required areacomparisons with the model.

After each comparison the enlarged or reduced edge direction image isagain subjected to a procedure for determining similarity indices asdescribed above in step 41.

From the four sizes of the edge direction image, 6 a, 6 b, 6 c, and 6 d,four sets of similarity indices are derived, each of which includes theentire image. In this case the size of the model has not been changed.As a further procedure example—not shown—FIG. 2 can also be processed insuch a way that the size of the edge direction image remains unchanged,but the size of the model is increased or decreased stepwise for eachiteration of procedure step 41. In this way further sets of similarityindices are obtained which can be used by step 42, position 60, as wellas size 60 a for determining the sought-for body part.

In addition, a threshold value is used if it is uncertain whether thetested target image 5 actually does contain the sought-for body part, aface in this case. Model 30 is an edge direction model of the type“face”, and it is only suitable for comparing faces as edge directionmodel with faces as edge direction image, developed from the grayscaleimage 5. The provision of a threshold value in step 42 ensures that thedetection process is meaningful even without foreknowledge about thepresence of a face. If a threshold value is not supplied, then, in thecase of a procedure without foreknowledge, it would not be possible todecide whether a maximum value found amongst the similarity indices is ameaningful agreement factor or not. However, in step 42 a thresholdvalue is not required when foreknowledge is available and thedetermination and reporting of the position and size of the expectedface is the actual purpose.

The operation of the procedure according to FIGS. 1 and 2 should beclear from the sequence of FIGS. 2 a to 2 e. The single procedure stepsare reflected like the initial image 5 which is represented in FIG. 2 aas a grayscale image with intensity variation subjected to acorresponding influence of the above-mentioned calculations. In FIG. 2 aseveral sections of the image which appear in the later images, aredrawn in and numbered similarly for comparison purposes.

Some of these image sections in the grayscale image are now explained.In general the background is indicated as 3; it consists of a slopingwall. There are two prominent lines 4 a and 4 b, which run diagonallybackwards behind the face image. The face is generally indicated with 2.The face and the shoulder section shown, exhibit inner “edges” in thesense of linear structures, indicated by 5 d and 5 e inside the face.Both shoulder areas which contrast strongly with the background 3, areindicated by 5 b and 5 c. The general borderline of the face is 5 awhich includes both edge or demarcation lines represented astwo-dimensional imaged edges 5 e and 5 d.

FIG. 2 b originates in procedure step 40; it is the preliminary step informing the edge direction information in the grayscale image forevaluation by means of model 30. At this stage the grayscale image hasbeen changed by an operator which increases the contrasts. The Sobeloperator, for example, is used for this purpose; it is able to providethe edge thickness and the edge direction for every pixel of the image.Because of the large magnification every pixel can be seen in FIG. 2 b,located in a generated grid. The shoulder edges 5 c and 5 b areespecially clear in this intermediate image of FIG. 2 b. The borderline5 a of the face is also obvious as well as borderlines 5 d and 5 e lyinginside the face. Background (3) lines 4 a and 4 b are also clearlyvisible as edges. Area 1 of the image contains little edge informationbecause there are hardly any edges. Having little information to beevaluated, this area must be regarded as noise.

The processing step for obtaining FIG. 2 b determines the describedexplanations of the edges by comparing pixel contrasts. The Sobeloperator can be regarded as a linear filter which operates like ahigh-pass on a flat plane. The grid shown, corresponds to the pixels.The linear clarity of the edges increases with the difference inintensity, for example edges 5 c and 5 b compared with background 3.

The actual vector representation of the image as stored in the computer,is not illustrated. It can however be represented immediately by meansof FIG. 2 b. A vector having direction and length is assigned to eachpixel. These vectors can be stored in a set of Cartesian coordinates, orin polar coordinates, using magnitude and angle. Every pixel P_(xy) inthe x·y plane of the image thus carries the orientation and length ofsuch a vector. This implies that every pixel has a measure for the edgedirection and a measure for the edge clarity.

In a further image, FIG. 2 c, all information which cannot contribute tomeaningful evaluation processing, is blanked out. In addition, athreshold value is used for comparison with each pixel in regard to edgeclarity. If the edge clarity of a pixel exceeds the threshold value, itis retained together with the direction information of this pixel in theimage of FIG. 2 c. Areas like section 1 or the dark part of thebackground 3 below line 4 b down to the very conspicuous edge 5 c, areblanked out or provided with a black value, namely zero. For a grayvalue scale running from 0 to 255, the latter indicates a prominent edgeand a bright value, while the value 0 is not processed. Apart from thecomparison with the threshold value, every pixel with a larger clarityindex, also has an analogue value of between 0 and 255.

In FIG. 2 d these analogue values have been deleted and the image is nowdigitized (in binary) as intended in step 40 of FIG. 1. All analogueinformation has now been eliminated and pixels are evaluated withrelation to the threshold value—whether it is exceeded or not. Edgeclarities lying above the threshold value, are evaluated as level 255(pertaining to one pixel). Values below or equal to the threshold valueare evaluated as zero. The image in FIG. 2 d can be regarded as adigitized edge image.

This is a result of using threshold value imaging. A direct transitionfrom FIG. 2 b to FIG. 2 d is possible if image 5 is processedpixel-by-pixel in step 40 in terms of comparison of threshold values anddigitizing. Digitizing causes a maximization of contrasts for furtherprocessing leading up to FIG. 2 d.

Edge direction information is obvious in FIG. 2 e, being represented byshort line segments. The direction of the line segments correspond to apolar coordinate representation in terms of direction. When closelyscrutinized, one can see that the directional information in the faceand shoulders correspond to the lines drawn in FIG. 2 a. Lines 5 c and 5b are clearly recognizable. The same holds for the inner lines 5 e and 5d. Even line 5 a, being the boundary of the face, is recognizable.Diagonal lines 4 a and 4 b are not so clear, but this is arepresentation problem of the coarse grid used for pixel 6. However,vertical and horizontal lines are especially clear.

FIG. 3 is the result of processing an image with reduced informationcontent. Only those areas of FIG. 2 e which entail essential edgeinformation, are included. The blanking out of unessential areas iseffected by coalescing the images of FIGS. 2 d and 2 e. Those areaswhich have been made black because of being below the threshold value,are also black in FIG. 3. The information content, including the edgeclarity and the edge direction, have been extracted from FIG. 2 e—onlythe directional information associated with pixels having a clarityindex above the threshold value.

The threshold value can be selected freely, depending on the type ofimage or the available contrasts. It can also fall away entirely if theimage as such exhibits little noise.

FIGS. 4 and 5 are direct opposites of FIGS. 3 and 2 e. FIG. 3 is theinverse of FIG. 4, while FIG. 5 corresponds directly with FIG. 2 e.

In FIG. 4 the total clarity information is neutralized, meaning, forexample, that all vectors are of equal length. Only the directionalinformation is retained. In the polar coordinate representation allpixels in the image possess a vector of equal length according to FIG.4, but a directional vector corresponding to the edge direction at thelocality of the pixel. Since the magnitude of the vector is omitted fromfurther evaluation, it can also be put equal to zero, retaining only thedirection for every pixel. The length of the directional vectors aregiven a standard size of 6 pixels.

The moment for standardization is determined by the actual version ofthe evaluation program, since it cannot take place during thepreliminary steps when the clarity information is still being processed.It implies that FIG. 2 e can be obtained directly from FIG. 2 a by usingthe Sobel operator mentioned above. FIG. 2 e can also be obtained alonga parallel route via FIGS. 2 b, 2 c, and 2 d. In this case the size ofthe vector can immediately be set to a standard value, six in this case,as drawn in the figure. These pixels can be recognized by the diagonalvector lines.

Evaluation of the now reduced information follows from FIG. 4 (or FIG.3). The information is reduced for areas of high clarity where onlydirectional information is retained. Theoretically additionalinformation can also be stored, but such information is not evaluated inthe next steps.

Then comparison with model 30, the model image, follows. At first it isassumed that a model which contains directional information, does existand represents a certain type of body part sought for in the edgedirection image with reduced information, namely FIG. 4. An example ofsuch a model appears in FIG. 8. The model 30 image drawn there containsdirectional information which has been reduced in a similar way. Theorigin of the model is explained below.

In procedure step 41 of FIG. 1 model 30 is compared with the imageobtained for FIG. 4. It should be assumed that the model is smaller thanthe image. The model may, for example, comprise only the area of thehead, while FIG. 4 contains other sections as well. The model is placedin the upper left corner of FIG. 4 and compared pixel-by-pixel with thereduced edges image. Then only directional information is compared.

The comparison can be structured in such a way that angle differencesare compared for every pixel position. A difference matrix can beconstructed by summing the angle differences for all pixels. The maximumnumber of terms in this angle difference sum is equal to the totalnumber of pixels in the model. This sum is stored at a positionrepresenting the upper left corner of the image. Subsequently the modelis shifted one pixel to the right and an angle difference sum is againdetermined for the entire model. This similarity index is stored next tothe previous pixel. This step is repeated until the model has coveredall pixel positions in the target image of FIG. 4 and all similarityindices have been calculated and stored.

It is obvious that the similarity values indicate a good agreement whenthe angular differences are small. The sum of small angle differences isalso small, meaning that the smallest index indicates the bestagreement. This is the purpose of the evaluation procedure.

Consider an example. If a model measures 40 pixels vertically and 30horizontally and a target image with reduced information contentmeasures 120 by 120 pixels, it means that a total of 80×90 positionscover all pixels of the image resulting in 7200 similarity values.

A similarity index can be represented by a brightness value. A highbrightness index of 255 indicates a low similarity, and a highsimilarity (a small sum of angle differences) corresponds to a low valueclose to zero, or a black point. FIGS. 6 and 6 a are an examplecorresponding to the image from FIG. 7, processed according to FIGS. 2 aup to 2 e. Before discussing this processing, the calculation of angledifferences is now explained.

It may happen that no overlapping can be determined when both model 30and the target image (FIG. 4) possess a reduced information content andboth compared pixels have no directional information, then the angledifference is set at a maximum to indicate a dissimilarity.

When angles and angle differences between 0° and 180° are accepted, thenthe maximum value is 180°. For a good agreement the angles of a pair ofpixels being compared, are equal and the difference is zero. Instead ofan angle difference a trigonometric function which is not multivaluedbetween 0° and 180°, can be used. The function (1−cos a) gives a valueof 0 for identical angles where a represents an angle difference, andwhen the angle difference is a maximum, a similarity index of 1 isobtained. When such a function is used instead of the plain angledifferences, the summation is reduced to terms lying between 0 and 1.

A formula for the direction difference d (angular agreement) at a pointis given under (1) below. When the result is naught (=0), the agreementbetween the directions is optimal for that pixel.d=sin(|f _(m) −f _(p)|)   (1)where

-   f_(m)=edge direction at a point of the model-   f_(p)=edge direction at a point in the edge oriented target image.

Instead of using angles in the range 0° to 180° only acute angles lyingbetween 0° and 90° can be used. Then values between 90° and 180°0 areconverted to acute angles by subtraction from 180° with the result thatan angle of 91° corresponds to 89°, and 180° indicates a good agreementlike 0°. Or the model as such can be set up in such a way that it onlycontains information about acute angles, then the comparison procedureis simplified, since sine and tangent formulas can be used in additionto cosine functions.

In the example described above, the sizes of the image and the modelresult in 7200 angle difference sums (similarity indices) forming a newmatrix which is represented in FIG. 6 on an m1 plane. Plane m1corresponds with the plane in the image of FIG. 7. This modified imageis now evaluated for a minimum to determine the position of the model onthe target image of FIG. 4 where the best agreement will be found. Forthis purpose all 7200 points have to be investigated to find a minimum.And for a minimum value to make sense, a threshold value is required ifthere is no previous knowledge about the presence of a sought-for facefor example. It is also possible that several minima can be found ifthere are several faces in the target image. Positional information canbe given based on the position of the pixel where a minimum is found.Depending on whether the agreement value occurs in the upper left cornerof the image or elsewhere, a simple calculation can give the exactlocation of model 30 in the target image. This result is obtained eitherby specifying its center and size, or the positions of the cornerpoints.

The range of summation of the single angle differences (or atrigonometrical conversion) can be decreased in the case of a reducednumber of similarity data (angle differences) and if a respectivesimilarity for a local allocation of model image and edge imageaccording to FIG. 4 is used. For example, the best applicable values arefound. The inferior values should not be used, since they might resultin deterioration of the similarity value. It could thus happen that outof 120 available values, for example, only 80 are used. It is thenpossible to use only ⅔ of the available values for summation and forobtaining a similarity index using only the smallest values in the sensedescribed above. Minimum values produce the best similarity indices. Ifa suitable inversion program is available, then everything can beinverted correspondingly.

It should be mentioned that this difference formation cannot be obtainedby means of a scalar product nor by using polar coordinates, as shownabove.

By selecting the model the type of body part to be sought for in thetarget image, can be determined.

If size information is given in addition to position information, thenthe model can be changed in size relative to the target image of FIG. 4.Alternatively, FIG. 4 can also be enlarged or reduced when the size ofmodel 30 is not changed. The procedure described above is executed forevery selected model size or for every selected target image size. Forseveral iterations several images m1, m2, m3, m4, m5, m6, etc. areobtained according to FIG. 6, where each of these images represents onemodel size and an image of all similarity indices of the respectivecomparisons of the model with the target image. Model size can bechanged in small steps, for example with a factor lying between 1.1 and1.4, and preferably between 1.2 and 1.3 to avoid large jumps insimilarity values.

The initial image of FIG. 7 is firstly evaluated according to theprocedure described above without the drawn-in frames 60, 73, 71, 72,and 70. The height of the grayscale image is h, and its breadth is b. Amatrix of similarity indices has been determined for this imageaccording to the procedures described above and as visualized in FIG. 6a for the first partial image m1. Several white positions are clearlydiscernible here, indicated with 59. In these image areas the leastagreement has been found.

After establishing the first similarity image m1, an enlarged model isused, leading to matrix m2 which exhibits a dark spot in area 58. Thegrid in FIG. 6 a reflects the relative positions for which similarityindices have been determined. Every pixel corresponds to a certainposition of the model relative to the edge direction image according toFIG. 4, but in this case it is restricted to another representation ofFIG. 7.

A further enlargement of the model results in matrix m3, where position60 becomes even more obvious. In this case the best agreement is foundwhere the smallest value of angle sums occurs and thus results in a darkspot with a value close to zero. In FIG. 6 this area is drawn in likeframe 60 in FIG. 7, where the sought-for face is located. Furtherincreases in the size of the model result in the sequence of matricesm4, m5, and m6. The stepwise black borders are explained because anincreasing model has fewer resulting points when the size of the targetimage, b×h, stays the same. This is valid for both vertical andhorizontal directions.

The frames drawn in FIG. 7 are different frames originating from severalpixels close to the minimum value. Frame 60 characterizes both theposition and the size of the facial part found, where the size is foundfrom the model which produced the resulting matrix m3 and the positionis given by the dark position 60 in this matrix.

Formation of the model was mentioned previously, but it was kept inreserve. It can in any case be obtained in the same way as the image ofFIG. 4. For the formation of the model, the threshold mentioned theremust not be zero. The data is digitized and only directional informationfor the model is stored. Then the model can be formed from a singleimage. But according to FIG. 8 the model can also be used in manyimages, 19 to 28, which comprise a sample. All these images form naturalfaces, used here as example of any desired body part.

Each of these images originated in this described procedure in a similarway as the image in FIG. 4. These images are combined in step 29 wherethe model is generated. It can be done by means of a middle value basedon the angles encountered at the pixel positions involved. To enablecomparison of these images, characteristic locations like eyes, nose andmouth should be given at similar positions in so far as the similarityof the size relationships of the images are given.

The range of applicability of the methods described covers the findingof faces as well as determining whether a face is at all present in thetarget image. Instead of a face, other body parts, like hands, legs, orentire persons, can also be located. This position can be found in thecase of static images as well as moving images because computation isfast enough. In moving images an object can be identified in terms ofits motions. In the case of motion of the sought-for object in a remoteor a nearby direction, size information can also be supplied.

1. Procedure for recognizing types of physically demarcated body parts(2) like faces, hands, and legs of a person (1) portrayed in a targetimage (5) when at least one, not yet recognized, body part (2) appearsin the target image in front of a background (3) and its position isindicated by means of positional information obtained and storedappropriately; characterized by (a) the target image (5) being convertedinto at least one intermediate image (FIGS. 2 b and 2 c) on which edgelines (5 a, 5 b, and 5 c) are accentuated, to be evaluated in afollowing step in regard to their direction (FIG. 2 e, 4); (b) preparinga model (FIG. 8, 30) based on other edge line information obtainedpreviously. The directions of the edge lines in the model are retainedfor determining which other model or which other edge line informationcorresponds to the sought-for type of body part (2); where only edgeline directions of a body part image are evaluated in comparison withthe model (30) to determine whether the body part image (2) appearing infront of the background (3) corresponds to the sought-for body part andwhether the accompanying positional information should be determined andstored;
 2. Procedure according to claim 1 when images of several bodyparts are present and can be consecutively compared with the sought-fortype according to their edge line information.
 3. Procedure according toclaim 1 when the positional information contains data about the positionof the sought-for body part in the target grayscale image (5). 4.Procedure according to claim 1 when both the first edge lines (5 d, 5 e;5 d′, 5 e′) inside the one (at least) body part image (2) as well as theedge line (5 a, 5 a′) of the physically demarcated body part (2) (atleast one) are used.
 5. Procedure according to claim 1 when a scenicpicture is represented which has various locally unchanging intensitiesand appears as a static image in digital form being a pixel-basedrepresentation containing intensity steps.
 6. Procedure according toclaim 1 when a scenic image has locally changing intensities and shows asubject (1) moving in front of the background (3) in such a way that therelative position or orientation of at least one depicted body partchanges with time.
 7. Procedure according to claim 1 when boundary lineinformation is seen as edge information which contains data about theorientation, especially if the three-dimensional edges of subject (1)contains lines shown in the two-dimensional intensity image (5). 8.Procedure according to claim 1 when the accentuation of the borderlinesis obtained from an evaluation of the intensity difference (contrast)between two adjacent image areas, especially by using a Sobel operatoras a type of high-pass plane filter.
 9. Procedure according to claim 1when the type of body part given by Model (30) is found in front of thebackground (3) in image (5) and made available for finding a binaryresult, namely whether at least the one type of body part correspondingto the model is to be found in the image or not (detection). 10.Procedure according to claim 1 when the grayscale image (5) is a colorpicture or one with colored components, in visible light or in aninvisible wavelength range.
 11. Procedure according to claim 1 when sizeinformation is determined and stored in addition to the positionalinformation.
 12. Procedure according to claim 11 when size informationis made available by changing the relative sizes of the model and thetarget image (5) and compare them for every pair of sizes to determinesimilarity values as well as selecting the best similarity index fordetermining size information.
 13. Procedure according to claim 1 whenthe intermediate image (FIG. 2 e, FIG. 4) has been evaluated in terms ofdirection compared with the model (30) by constructing several sums ofangle oriented similarity values, especially the formation of a sum ofangle differences between borderline sections of the model and of theintermediate image (FIG. 2 e, FIG. 4) evaluated in regard to direction.14. Procedure according to claim 13 when a threshold value is madeavailable which is compared to all similarity values distributed overthe target image, only concluding that a sought-for body part (2) ispresent in the target image if at least one index is below the minimumthreshold value.
 15. Procedure for the detection of physicallydemarcated body parts like a face, a hand or a leg of a person's image(5) when a body part (2) is depicted in front of a background (3);characterized by borderlines (5 d, 5 e) of the image are only evaluatedalong directions (5 a′, 4 a′, 4 b′, 5 c′) to determine by comparisonwith model (30) whether the body part image corresponds to a typepresented by the model, when at least borderline directions (5 d′ and 5e′) inside at least one body part image and borderline directions (5 a)of at least one physically demarcated body part image are used forlocating its position and the corresponding positional information isstored.
 16. Preparation procedure for the detection of physicallydemarcated body parts like a person's face, hand, or leg in an image (5)when a body part (2) is depicted in front of a background (3) anddirectional information of the borderlines of at least one body partimage and its background for determining in a later step whether thebody part image corresponds to a type given by model (30). In preparingfor a detection procedure a working image is constructed by calculationfor which (a) the image (5) is converted into another representation(FIG. 2 b) which indicates directional data for the borderlines with aclarity value given for each direction; (b) the further representationis processed by using a threshold value (FIG. 2 c) to repressdirectional data at positions where the clarity is below the thresholdvalue, resulting in reduced directional information (5 a′, 5 b′) in theworking image (FIG. 3, FIG. 4); to enable the working image to be usedas model (30) or as target image (FIG. 4).
 17. Procedure according toclaim 16 when the remaining directional information stays unchanged inthe working image, but the clarity indices are set at a standard valuefor all remaining directions to neutralize the clarity information. 18.Procedure according to claim 17 when the remaining information (theremaining directions) is evaluated in the next step by using model (30)and the position of the body part concerned is located and thepositional information is stored.
 19. Procedure according to claim 16when the plane of the representation which has been processed withthreshold values, is now processed with a binary operator to obtain apattern image (FIG. 2 d) which is then combined with the nextrepresentation.
 20. Procedure according to claim 13 when only a reducednumber of terms are used to obtain several sums of similarity indicesfor each relative position of model (30) and the intermediate image(FIG. 4) which has been evaluated according to direction, especiallybelow 80% of the available summation terms for every relative positionwhen only those similarity data having the best similarity indices areused.
 21. Procedure according to claim 13 or claim 20 when only acuteangles are compared; angles between 90° and 180° are converted tocorresponding values between 90° and 0° before being compared. 22.Procedure according to claim 21 when model (30) contains onlydirectional angles between 0° and 90° for borderlines in the model. 23.Procedure for locating at least one physically demarcated body part likea face, a hand, or a leg in an image of a person (i) when at least onebody part (2) in the image (5) is depicted in front of a background (3);(ii) when borderlines (5 d, 5 e) are accentuated (FIG. 2 b), and atleast the directions (5 a′, 4 a′, 4 b′, 5 c′) of some of theseemphasized borderlines (FIG. 4) have been evaluated by repeatedcomparisons with model (30) where each comparison occurs at anotherrelative local placement of model (30) and the emphasized borderlines(FIG. 4) to obtain a similarity index for every location; Selection of abest similarity index from the values obtained in the comparisons; thisdetermines and stores the position of the body part image.
 24. Procedureaccording to claim 23 when the part of the accentuated borderlines (FIG.4) forms an intermediate image and the similarity value of a localcorrespondence is determined by the summation of the similarity dataoccurring at the borderline angles, especially by constructing a sum ofangle differences between borderline sections of model (30) and theintermediate image (FIG. 2 e, FIG. 4) over at least a significant areaof model (30); or by finding angle differences between borderlinesections of model (30) and the intermediate image (FIG. 2 e, FIG. 4)with the corresponding trigonometrical conversions and summation of theresulting similarity data for the similarity index.
 25. Procedureaccording to claim 23 when a threshold value is available and comparedwith all the similarity indices distributed over the plane of the image(5). The presence of a sought-for body part (2) is only confirmed if avalue smaller than the threshold minimum is found at least one position.26. Procedure according to claim 23 when a similarity value isdetermined for a local correspondence only when using thoseangle-oriented similarity data for which both model (30) and theintermediate image (FIG. 2 e) a directional value is available at thesame position and there is no angle (directional information) in atleast one of the two images (FIG. 2 e, 30), then this contributesnothing to the similarity index at this position.
 27. Procedureaccording to claim 24 when a smaller similarity value means a better orgreater similarity.